Continuous-Time Attention for Sequential Learning

Attention mechanism is crucial for sequential learning where a wide range of applications have been successfully developed. This basically trained to spotlight on the region interest in hidden states sequence data. Most attention methods compute score through relating between query and discrete-time state trajectory represented. Such could not directly attend continuous-time which represented via neural differential equation (NDE) combined with recurrent network. paper presents new method tightly integrated NDE construct an attentive machine. The performed at all times over different kinds irregular time signals. missing information data due sampling loss, especially presence long sequence, can be seamlessly compensated attended representation. experiments samples from human activities, dialogue sentences medical features show merits proposed activity recognition, sentiment classification mortality prediction, respectively.